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Abstract — Compressed sensing has shown that it is possible 
to reconstruct sparse high dimensional signals from few linear 
measurements. In many cases, the solution can be obtained 
by solving an £i -minimization problem, and this method is 
accurate even in the presence of noise. Recent a modified 
version of this method, reweighted £i -minimization, has been 
suggested. Although no provable results have yet been attained, 
empirical studies have suggested the reweighted version outper- 
forms the standard method. Here we analyze the reweighted 
^1 -minimization method in the noisy case, and provide provable 
results showing an improvement in the error bound over the 
standard bounds. 

I. Introduction 

Compressed sensing refers to the problem of realizing a 
sparse input x using few linear measurements that possess 
some incoherence properties. Its applications range from error 
correction to image processing. Since the measurements are 
linear, the problem can be formulated as the recovery of a 
signal a; e M'' from its measurements u — where $ is 
a m X d measurement matrix. In the interesting case where 
m <C d, it is clearly impossible to reconstruct any arbitrary 
signal, so we must restrict the domain to which the signals 
X belong. To that end, we consider sparse signals, those with 
few non-zero coordinates relative to the actual dimension. In 
particular, for s ^ d, we say that a signal x G M"^ is s-sparse 
if X has s or fewer non-zero coordinates: 

llxllo = \{t : X, + 0}| < s. 

It is now well known that many signals such as real-world 
audio and video images or biological measurements are sparse 
either in this sense or with respect to a different basis. 

Much work in the field of compressed sensing has led to 
promising reconstruction algorithms for these kinds of sparse 
signals. One solution to the recovery problem is simply to 
select the signal whose measurements are equal to those of 
X, with the smallest sparsity. That is, one could solve the 
optimization problem 

min llxllo subject to ^x — $a;. (Lq) 

This straightforward approach is quite accurate, and if the 
columns of the m x d matrix <& are in general position, it 
can recover signals that are up to m/2-sparse. The crucial 
drawback to the problem (Lq) is of course that it is highly 
nonconvex and requires a search through the exponentially 
many column sets. This clearly makes it of little use in 
practice. A natural alternative then is to relax the problem 
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(Lo), and use a convex problem instead. One can then consider 
instead the -minimization problem 

min subject to ^x — ^x. (Li) 
Here and throughout, || • ||i denotes the standard £i-norm: 

Problem (Li) is convex, and can actually be reformulated 
as a linear program. Due to recent work Unear programming 
and smoothed analysis, it is now well known that it can be 
solved efficiently in practice [1], [11]. Notice that the solution 
to {Li) is the contact point where the smallest ^i-ball meets 
the subspace x + kcr $. The geometry of the octahedron 
lends itself well to sparsity due to its wedges at the lower 
dimensional subspaces. 

Indeed, Candes and Tao prove that when the measurement 
matrix $ satisfies a certain quantitative property, the solution 
to the problem (Li) will be the original sparse signal ([3], see 
also [10]). This restricted isometry condition guarantees that 
every m x s submatrix of $ approximately preserves norm: 

Definition 1.1: The measurement matrix $ satisfies the 
restricted isometry condition with parameters (s, 5) if 

{l-5)\\x\\l<\\^x\\l<[l + 5)\\x\\l 

holds for all s-sparse vectors x. Here and throughout, || • |j2 

denotes the usual Euclidean norm: ||a;||2 == (SiLi ^f)^^^- 

It has been shown that many random measurement matrices 
satisfy the restricted isometry condition with small 6 and 
m nearly linear in the sparsity. In [7] it is shown that 
measurement matrices whose entries are subgaussian satisfy 
the restricted isometry condition with parameters (s, 5) with 
high probability when 

Note that this implies in particular that matrices whose entries 
are (normalized) random Gaussian or Bernoulli satisfy the re- 
stricted isometry condition with this number of measurements. 
An alternative type of measurement matrix is a partial bounded 
orthogonal matrix. One such example is obtained by selecting 
m rows uniformly at random from the discrete Fourier matrix. 
Rudelson and Vershynin show in [9] that these matrices satisfy 
the restricted isometry condition with parameters (s, 6) with 

Note that in both cases we need only ra k, s log d measure- 
ments. 

Candes and Tao showed that if the measurement matrix 
$ satisfies the restricted isometry condition with parameters 



(3s, 0.2), that the s-sparse signal x is the unique solution to 
the problem (ii). In [5], Candes sharpened these results to 
show success with parameters just (2s, -\/2 — 1). 

As is evident, these results provide strong guarantees for 
the ^1 -minimization problem on sparse signals. In practice, 
however, signals are rarely exactly sparse, and may often be 
corrupted by noise. The problem then becomes to reconstruct 
an approximately sparse signal x from its noisy measurements 
u = ^x + e, where e is an error vector In this case, 
problem {Li) will clearly not suffice for recovery, since with 
noise the original signal may not even satisfy the constraint 
requirements. However, the problem can simply be adapted to 
account for noise error: 

min subject to \\^x — u\\2 < s, (L[) 

where e is a noise parameter with ||e||2 < e. Candes, Romberg, 
and Tao showed that the solution to the problem (L'l) is 
close in Euclidean norm to the original signal x [2]. Candes 
improved these results to provide the following. 

Theorem 1.2 (li-minimization from [5]): Assume $ has 
52s < V2 — 1. Let x be an arbitrary signal with noisy mea- 
surements + e, where ||e||2 < s. Then the approximation 
i to X from £i -minimization satisfies 

||x-x||2<Ce + C7'fc^, 

where C ^ C = p = and a = . 

The error bound provided here is optimal up to the con- 
stants, as the error e + can be viewed as the un- 
recoverable energy due to the inherent noise. See [8] for a 
detailed discussion of the unrecoverable energy. Note also that 
in the case where x is exactly sparse, but the measurements 
are noisy, the error bound Ce is proportional to the norm of 
the error vector e. Although these results provide very strong 
guarantees, recent work has been done on a variant of the £i- 
minimization problem that seems to outperform the standard 
method. 

II. Reweighted ^1-MINIMIZATION 

As discussed above, the £i -minimization problem (Li) is 
equivalent to the nonconvex problem (Lq) when the measure- 
ment matrix $ satisfies a certain condition. The key difference 
between the two problems of course, is that (Li) depends 
on the magnitudes of the coefficients of a signal, whereas 
(Lq) does not. To reconcile this imbalance, a new weighted 
£i -minimization algorithm was proposed by Candes, Wakin, 
and Boyd [4]. This algorithm solves the following weighted 
version of (Li) at each iteration: 

d 

min SiXi subject to ^x — ^x. (WLi) 

l — l 

It is clear that in this formulation, large weights Si will 
encourage small coordinates of the solution vector, and small 
weights will encourage larger coordinates. Indeed, suppose the 
s-sparse signal x was known exactly, and that the weights 
were set as 6i = j-i-r. Notice that in this case, the weights are 



infinite at all locations outside of the support of x. This will 
force the coordinates of the solution vector x at these locations 
to be zero. Thus if the signal x is s-sparse with s < m, these 
weights would guarantee that x ~ x. Of course, these weights 
could not be chosen without knowing the actual signal x itself. 
However even if the weights are close to the actual signal, the 
geometry of the weighted £i-ball becomes "pinched" toward 
the signal, decreasing the liklihood of an inaccurate solution. 

Although the weights might not initially induce this geom- 
etry, one hopes that by solving the problem (WLi) at each 
iteration, the weights will get closer to the optimal values 
jj-j, thereby improving the reconstruction of x. Of course, one 
cannot actually have an infinite weight, so a stability parameter 
must also be used in the selection of the weight values. The 
reweighted £i -minimization algorithm can thus be described 
precisely as follows. 



In [4], the reweighted £i -minimization algorithm is dis- 
cussed thoroughly, and experimental results are provided to 
show that it often outperforms the standard method. How- 
ever, no provable guarantees have yet been made for the 
algorithm's success. Here we analyze the algorithm when the 
measurements and signals are corrupted with noise. Since the 
reweighted method needs a weight vector that is somewhat 
close to the actual signal x, it is natural to consider the noisy 
case since the standard £i -minimization method itself produces 
such a vector. We are able to prove an error bound in this 
noisy case that improves upon the best known bound for the 
standard method. We also provide numerical studies that show 
the bounds are improved in practice as well. 

III. Main Results 

The main theorem of this paper guarantees an error bound 
for the reconstruction using reweighted -minimization that 
improves upon the best known bound of Theorem 1 1.21 for the 
standard method. For initial simplicity, we consider the case 
where the signal x is exactly sparse, but the measurements u 
are corrupted with noise. Our main theorem. Theorem 13. II will 
imply results for the case where the signal x is arbitrary. 



Reweighted -minimization 

Input: Measurement vector u e W\ stability parameter 

a, noise parameter e 

Output: Reconstructed vector x 

Initialize: Set the weights Si = 1 for i = 1 . . .d. 

Repeat the following until convergence or a 

fixed number of times: 

Approximate: Solve the reweighted £i- 
minimization problem: 

d 

X = argminy~^ SjXj subject to ||$a: — u||2 < £• 
Update the weights: 

\Xi\+ a 



Theorem 3.1 (Reweighted ii, Sparse Case): Assume $ 
satisfies the restricted isometry condition with parameters 
(2s, S) where 6 < \/2 — 1. Let x be an s-sparse vector with 
noisy measurements u — ^x + e where ||e||2 < e. Assume the 
smallest nonzero coordinate fi of x satisfies fi > Then 
the limiting approximation from reweighted £i -minimization 
satisfies 

ll^-^lh < C"e, 



^ and a 



1-5 • 



where C" 
Remarks. 

1. We actually show that the reconstruction error satisfies 

2as 



i 2< 



1 



4ae 



4a£p 



(III.l) 



This bound is stronger than that given in Theorem l3.1l which is 
only equal to this bound when /i nears the value ■ However, 



the form in Theorem 13.11 is much simpler and clearly shows 
the role of the parameter S by the use of p. 

2. For signals whose smallest non-zero coefficient /i does 
not satisfy the condition of the theorem, we may apply the 
theorem to those coefficients that do satisfy this requirement, 
and treat the others as noise. See Theorem 13.21 below. 

3. Although the bound in the theorem is the limiting bound, 
we provide a recursive relation dlll.SI l in the proof which 
provides an exact error bound per iteration. In Section |IV] we 
use dynamic programming to show that in many cases only a 
very small number of iterations are actually required to obtain 
the above error bound. 

We now discuss the differences between Theorem 11.21 and 
our new result Theorem l3.1l In the case where i5 nears its limit 
of — 1, the constant p increases to 1, and so the constant 
C in Theorem 11.21 is unbounded. However, the constant C" 
in Theorem 13 . 1 1 remains bounded even in this case. In fact, as 
5 approaches -\/2 — 1, the constant C" approaches just 4.66. 
The tradeoff of course, is in the requirement on fi. As 6 gets 
closer to \/2 — 1, the bound needed on ii requires the signal 
to have unbounded non-zero coordinates relative to the noise 
level e. However, to use this theorem efficiently, one would 
select the largest 5 < V2 — 1 that allows the requirement on 
p to be satisfied, and then apply the theorem for this value of 
6. Using this strategy, when the ratio j = 10, for example, 
the error bound is just 3.85e. 

Theorem l3.1l and a short calculation will imply the following 
result for arbitrary signals x. 

Theorem 3.2 (Reweighted Assume <!> satisfies the re- 
stricted isometry condition with parameters (2s, \/2 — !)■ Let 
X be an arbitrary vector with noisy measurements u — $a; + e 
where |je||2 < e. Assume the smallest nonzero coordinate p 
of Xs satisfies p > ^z^, where £o — l-2(||a; — Xs||2 + -^||a; — 
Xslli) + e. Then the limiting approximation from reweighted 
£i-minimization satisfies 



\\x~xh< 



4. la /||a: - 



and 



i||2 < 



2.4q! 



a;^ 2 + 



Xs\\\ 



where p and a are as in Theorem 13. II 

Again in the case where b nears its bound of \/2 — 1, 
both constants C and C" in Theorem 11.21 approach infinity. 
However, in Theorem 13.21 the constant remains bounded 
even in this case. The same strategy discussed above for 
Theorem 13.11 should also be used for Theorem 13.21 Next we 
begin proving Theorem 13.11 and Theorem 13.21 

A. Proofs 

We will first utilize a lemma that bounds the I2 norm of a 
small portion of the difference vector x — x hy the €i-norm 
of its remainder. This lemma is proved in [5] and essentially 
in [2] as part of the proofs of the main theorems of those 
papers. 

Lemma 3.3: Set h — x — x, and let a, e, and p be as in 
Theorem 13.11 Let Tq be the set of s largest coefficients in 
magnitude of x and Ti be the s largest coefficients of /it=. 
Then 



\\hTo 



UTi II 2 



< ae 



and 



1 



ll"(ToUTi)-l|2 < —^Whr-, 
y/S 

We will next require two lemmas that give results about a 
single iteration of reweighted £1 -minimization. 

Lemma 3.4 (Single reweighted £i-minimization): Assume 
$ satisfies the restricted isometry condition with parameters 
(2s, \/2 — 1). Let X be an arbitrary vector with noisy 
measurements u = ^x + e where ||e||2 < e. Let w be a vector 
such that ||w — x||oo < A for some constant A. Denote by Xg 
the vector consisting of the s (where s < |supp(a;)|) largest 
coefficients of x in absolute value. Let p be the smallest 
coordinate of Xg in absolute value, and set 6 = — XsHoo- 
Then when p > A and pCi < 1, the approximation from 
reweighted ^i-minimization using weights Si = l/{wi + a) 
satisfies 

\\X - XsWl 



(111.2) 



(III.3) 



X- 



where Di 



x\\2 < Die 

(l+Ci>^ jj 



Do 



Co 



2(A+a+b) 



C2 



(l + Cl)pC2 

1-pCi 



, Ci 



A+a+b 



, and p and a are as in Theorem 13.1 



Proof: Now we begin the proof of Lemma 13.41 
Set h and Tj for j > as in Lemma 13.31 For simplicity, 
denote by || • the weighted ^i-norm: 



1 



Wi 



Since x = x + h is the minimizer of ( |VF£i| l, we have 

\\x\\w > \\x + h\\^ = \\{x + h)To\\w + + h)T-\\^ 



\hn 



This yields 



< 



II^TolU + 2||xt-|| 



Next we relate the reweighted norm to the usual £i-norm. We 
first have 

\\hTA\^ 



by definition of the reweighted norm as well as the values of 
A, a, and b. Similarly we have 

l^lTolll 



'■To 1 1 ID 



< 



II — A + a 

Combining the above three inequalities, we have 

WhTsh < (A + a + b)\\hTs\Ua (in.4) 

< {A + a + b){\\hT,\\ro+2\\xTs\\w) 



<^^^\\hTo\\i + '2{A + a + b)\\xTj 

fj> — Ji. -T d 



(111.5) 



Using ( IIII.3I) and illlAi along with the fact ||/itoI|i < 
\/s|l/iTol|2, we have 

|IVoUTO=l|2 < Ci\\hT„\\2+C2\\xTs\\^, (111.6) 

where Ci = and C2 = . By dnLl of 

Lemma [33l we have 



||/iTouTi||2 <as+ —=\\hT-\\l, 
where p = and a = Thus by jlini i, we have 

\\hToUT^h<ae + -^{Ci\\hT,\\i + 2{A + a + b)\\xT§\\^) 
^as + pCi\\hToUTih + pC2\\xT§\\w 

Therefore, 

\\hT0UTA\2 < {l-pCi)-\as + pC2\\xTs\\w)- (III.7) 
Finally by ( |IIL6] | and jllLTl l. 

||^||2 < ll^iToUTi II2 + ||/i(ToUTi)<=||2 

< il + C\)\\hT„uT^\\2 + C2\\xTs\\u, 

<(i + c,)(^^^^)+c.iKH|,.. 

Applying the inequality lla;^^': < (l/a)||a;T=||i and simpli- 
fying completes the claim. 



To this end, let E{k) for k = 1, . . ., be the error bound on 
\\x — Xk\\2 where Xk is the reconstructed signal at the fc*'* 
iteration. Then by Theorem 11.21 and Lemma 13.51 we have the 
recursive definition 



2a 



E{1)^- e, E{k + 1) 

I - p 



(1 



1 



Ejk) 
i-E(k) 



(III.8) 



Here we have taken a — > iteratively (or if a remains fixed, 
a small constant 0{a) will be added to the error). First, we 
show that the base case holds, E{\) < E{2). Since p > j^, 



we have that 



E{1) 



2ae 
1-p 



p-E{l) p-j^^ 
Therefore we have 



< 1. 



tJ.-E(l)' 



1 



-£ < 



2a 



E(l) 



Next we show the inductive step, that E{k + 1) < E{k) 
assuming the inequality holds for all previous k. Indeed, if 

E(^k) < E{k - 1), then we have 



E{k + 1) = 



(1 



E{k) 



^-1 , E(k-l) ^ 

< ^ll±zE^tz2^e = E{k). 



1 



E{k) 
Pfj.-E(k) 



1 



E(k-l) 
P ti-E(k-l) 



Since p > and p < 1 we have that p — E{k) > and 

P f_i-^lk) — ^i^) is ^ls° bounded below by zero. Thus 
E(k) is a bounded decreasing sequence, so it must converge. 
Call its limit L. By the recursive definition of E{k), we must 
have 

n J 



L 



(1 + 77^)" 



1 



Pj^ 



Solving this equation yields 



L = 



p — \/ p^ — ipae — Apaep 



Applying Lemma 13.41 to the case where x — Xg =0 and 
6 = yields the following. 

Lemma 3.5 (Single reweighted ii-minimization, Sparse Case) 
Assume $ satisfies the restricted isometry condition with 
parameters (2s, ^/2 — 1). Let x be an s-sparse vector with 
noisy measurements u ~ ^x + e where ||e||2 < s. Let w 
be a vector such that ||w — a;||oo < A for some constant A. 
Let p be the smallest non-zero coordinate of x in absolute 
value. Then when p> A, the approximation from reweighted 
^1 -minimization using weights 5i ~ l/{wi + a) satisfies 

||a; - x\\2 < Die. 

Here Di = Ci = -^+° , and a and p are as in 

^ 1— pCi ^ fi—A-\-a ' " 



2(1 + P) 

where we choose the solution with the minus since E{k) is 
decreasing and E{1) < p/2 (note also that L = when e = 
0). Multiplying by the conjugate and simplifying yields 

Apae + Apaep 

— , ~ 

2(1 + p){p + — Apae — Apaep) 

2ae 



1 



Y 4Qg 

4a£ 



4a£p 



Then again since p > we have 

2ae 



L < 



1 



P 



Theorem 13.11 

Now we begin the proof of Theorem 13.11 Proof: The 
proof proceeds as follows. First, we use the error bound in 
Theorem 11.21 as the initial error, and then apply Lemma 13.51 
repeatedly. We show that the error decreases at each iteration, 
and then deduce its limiting bound using the recursive relation. 



Proof: Now we begin the proof of Theorem 13.21 By 
Lemma 6.1 of [8] and Lemma 7 of [6], we can rewrite <i>a; + e 
as $2:, + e where 



1 



PI2 < 1.2(||a; - a;.||2 + ^IN - + |le|l2 

< 2.04( 



+ l|e||2. 



This combined with Theorem 13.11 completes the claim 



IV. Numerical Results and Convergence 

Our main theorems prove bounds on the reconstruction error 
limit. However, as is the case with many recursive relations, 
convergence to this threshold is often quite fast. To show this, 
we use dynamic programming to compute the theoretical error 
bound E{k) given by ( IIII.8I 1 and test its convergence rate to the 
threshold given by eqrefactualbnd. Since the ratio between /i 
and e is important, we fix /i = 10 and test the convergence for 
various values of e and S. The results are displayed in Figure [T] 
We observe that in each case, as 6 increases we require slightly 
more iterations. This is not surprising since higher S means a 
lower bound. We also confirm that less iterations are required 
when the ratio fi/e is smaller. 



(a) Number of iterations til convergence, Nfl=10, e=0.01 (b) Number ot iterations tii convergence. M=10, e=0.1 
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5 value 5 value 

Fig. L The number of iterations required for the theoretical error 
bounds eqrefEk to reach the theoretical error threshold jllLlt when (a) 
l_i = 10, £ = 0.01, (b) ^ = 10, e = 0.1, (c) II = 10, e = 0.5, (d) 
/X = 10, £ = 1.0. 

Next we examine some numerical experiments conducted to 
test the actual error with reweighted -minimization versus 
the standard £i method. In these experiments we consider 
signals of dimension d = 256 with s = 30 non-zero entries. 
We use a 128 X 256 measurement matrix $ consisting of 
Gaussian entries. We note that we found similar results when 
the measurement matrix $ consisted of symmetric Bernoulli 
entries. For each trial in our experiments we construct an s- 
sparse signal x with support chosen uniformly at random and 
entries from either the Gaussian distribution or the symmetric 
Bernoulli distribution, all independent of the matrix <I>. We 
then construct the normalized Gaussian noise vector e, and run 
the reweighted .^i -algorithm using e such that e'^ — <j'^{m + 
2\/2m) where cr^ is the variance of the normalized error 
vectors. This value is likely to provide a good upper bound on 
the noise norm (see e.g. [2], [4]). We set a = fc/1000 in the fc*'' 
iteration. We run 500 trials for each parameter selection and 
signal type. We found similar results for non-sparse signals, 
which is not surprising since we can treat the signal error as 
measurement error after applying the measurement matrix (see 
the proof of Theorem 13. 2| i. Figures |2] and |3] display the results 
of the experiments and demonstrate large improvements in 
the error of the reweighted reconstruction x compared to the 
reconstruction x* from the standard method. 



Reconsfruction error for d=256 m=1 28 s=30 with decreasing 
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Fig. 2. Improvements in the £2 reconstruction error using reweighted £1- 
minimization versus standard l\ -minimization for Gaussian signals. Error plot 
(left) and histogram of improvement factors \\x — x\\2/\\x — x*\\2 (light). 



Rsconstmction error for d=256rri=12e e-30 witit rfecreasing a Irrrprovement by rewergtited LI after 9 iterations 

tord=256 m-128 s=30 v^ith decreasing e 




Fig. 3. Improvements in the £2 reconstruction error using reweighted l\- 
minimization versus standard £1 -minimization for Bernoulli signals. Error plot 
(left) and histogram of improvement factors \\x — x\\2/\\x — x*\\2 (right). 
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